Method and apparatus to use array sensors to measure multiple types of data at full resolution of the sensor

ABSTRACT

An actuator is configured to move a sensor array between first and second positions in order to provide color image data and other data with full resolution of the sensor array. In many embodiments, the output resolution of the sensor array for each type of data comprises twice the resolution of the sensor array without movement. The alternating movement of the sensor array between the first and second positions provides output images with decreased artifacts that might otherwise be present without the alternating movement of the sensor array.

CROSS-REFERENCE

The present application is a continuation of U.S. application Ser. No.14/306,844, filed Jun. 17, 2014, now U.S. Pat. No. 8,917,327 issued onDec. 23, 2014, which application claims the benefits U.S. ProvisionalApplication Ser. Nos. 61/961,031, filed Oct. 4, 2013; 61/961,821, filedOct. 25, 2013; and 61/925,339, filed Jan. 9, 2014, the full disclosuresof which are incorporated herein by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has notobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

Sensors are devices that measure values of data. The data can bemeasured as a single value via a single sensor, such as measuring avelocity vector, or in the form of multiplicity of measurements of thedata, such as measuring the intensity of light falling within a certainband or spectrum in a certain scene, or being emitted as a result ofcertain stimulus to an object, such magnetic resonance imaging (MRI).

Sensors can be in the form of a single-dimensional or two-dimensionalarray. Two dimensional array sensors have a multiplicity of “small”measuring areas, or “microsensors”. Traditionally, two-dimensional arrayof microsensors are fabricated to be sensitive to a certain type ofmeasurement; that is, the array is fabricated to measure a single typeof data. An example of sensor arrays that are formed from a multiplicityof microsensors is an everyday camera. Cameras contain microsensors,called pixels, that are sensitive to visible light, i.e. eachmicrosensor measures the intensity of light falling on its surface. Eachmicrosensor (or pixel) has a photo-electric area which is in charge ofcollecting incoming light and converting it to an electric signal thatis a function of the intensity of that light. Therefore, when the lightfalls on each of the microsensors, the light is converted to anelectrical signal that is read as an analog value. The choice of themicrosensor whose value is to be read is done by using a decodingcircuitry which chooses the row and column connected to themicrosensors. The value is amplified, converted to a digital value, andstored in a memory for subsequent use in a digital computing andprocessing system, which can be employed in different applications.

The array of pixels can be made sensitive to different colors, byputting different light-admittance filters, one on every pixel. Forexample, one filter can admit light in the green color spectrum, anotherin the red-color spectrum, and a third in the blue-color spectrum. TheBayer pattern is a commonly used microsensor (pixel) layout for a twodimensional sensor array that are used in—visible spectrum—cameras.

Measuring two different types of data can be categorically divided intotwo philosophies: first, the straight-forward approach of using aseparate sensor for every measurement, one for grabbing visible images,and the other for grabbing infrared data (to be used for depthcomputation). The second approach is to use a mix between the pixelsthat measure visible light and the pixels that measure infrared. Thefirst approach has the disadvantage of employing a separate sensor. Thesecond approach has two major drawbacks. First, it has a loss ofresolution of both measurements since the pixel array is being spaceshared by different types of microsensors. Second where the microsensorof depth is put “under” the microsensors of visible light, a dramaticdegradation of the image quality of the visible image may occur.

These and other needs are addressed by the various aspects, embodiments,and/or configurations of the present disclosure.

SUMMARY OF THE INVENTION

The present disclosure is directed to an image processing system able tocapture multiple types of data. The teachings of this disclosure applyto charge coupled devices (CCD's) and complementarymetal-oxide-semiconductor (CMOS) technologies employed for light sensingdevices. In many embodiments, an actuator is configured to move a sensorarray between first and second positions in order to provide color imagedata and other data with full resolution of the sensor array. In manyembodiments, the output resolution of the sensor array for each type ofdata comprises twice the resolution of the sensor array withoutmovement. The alternating movement of the sensor array between the firstand second positions provides output images with decreased artifactsthat might otherwise be present without the alternating movement of thesensor array.

In one aspect, disclosed herein is an apparatus comprising: (a) a sensorarray comprising a plurality of Bayer patterns disposed on a firstplurality of pairs of adjacent linear arrays to sense Bayer patterndata, a second plurality of pairs of adjacent linear arrays havingdifferent patterns located between the first plurality of pairs ofadjacent linear arrays to sense a different type of data; (b) anactuator to move the sensor array from a first position to a secondposition, wherein physical positions of the first plurality of adjacentpairs of linear arrays alternate with physical positions of the secondplurality of pairs of adjacent linear arrays; and (c) circuitry coupledto the actuator and the sensor array to output the Bayer pattern dataand the different type of data from each of the first position and thesecond position. In some embodiments, each pixel of the Bayer patterndata is output associated with the first position or the second positionand wherein each pixel of the different type of data is outputassociated with the first position or the second position.

In some embodiments, the different pattern comprises one or more of aninfrared filter pattern, an ultraviolet filter pattern, a non-Bayervisible light filter pattern, or a pattern comprising no filter.

In some embodiments, each of the pairs of the first plurality of pairsof the adjacent linear arrays comprises two adjacent pairs and whereineach of the second plurality of pairs of adjacent linear arrayscomprises two adjacent pairs of linear arrays.

In some embodiments, each of the pairs of the first plurality of pairsof the adjacent linear arrays comprises three or more adjacent pairs oflinear arrays and wherein each of the second plurality of pairs ofadjacent linear arrays comprises three or more adjacent pairs of lineararrays.

In some embodiments, the apparatus further comprises a first limitswitch to signal when the sensor array is located in the first positionand a second limit switch to signal when the sensor array is located inthe second position, wherein the circuitry is configured to integratefirst composite data of the sensor array in the first position inresponse to the first limit switch sensing the sensor array moving intothe first position and integrate second composite data of the sensorarray in response to the second limit switch sensing the sensor movinginto the second position.

In some embodiments, the circuitry in the apparatus is configured togenerate the Bayer pattern data from the first composite data and thesecond composite data and to generate the different type of data fromthe first composite data and the second composite data.

In some applications, the circuitry comprising instructions to move thesensor array to the first position and measure first data of the sensorarray in the first position and move the sensor array to the secondposition and measure second data of the sensor array in the secondposition, the processor further comprising instructions to provide afirst full frame image from the first plurality of pairs of adjacentlinear arrays having the Bayer pattern and to output a second full frameimage from the second plurality of pairs of adjacent linear arrayshaving different pattern, wherein the sensor comprises a number ofpixels and wherein each of the first full frame image and the secondfull frame image comprising the number of pixels of the sensor array.

In certain embodiments, the actuator in the apparatus comprises amicro-electrical mechanical system.

In some applications, each of the plurality of Bayer patterns comprisesa red pixel to sense red light, a blue pixel to sense blue light, and apair of diagonal green pixels to sense green light. Additionally, thefourth sensor senses an infrared electromagnetic wave, and/or acircuitry is coupled to the moveable gate to control the motion of thegate.

In some embodiments, the apparatus further comprises digital datastorage to store an output generated from the sensor arrays.

In certain embodiments, the apparatus further comprises a digital signalprocessor to control one or more of the following: transferring theoutput to digital data storage, timing of the image acquisition,movement of the arrays, configuring the circuitry, configuring imageformation, and generating images.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 depicts a microsensor array; in this case, an array ofmicrosensors are arranged in a geometrical layout, a rectangular twodimensional array with equal interlacing of two types of themicrosensors, in accordance with embodiments.

FIG. 2 depicts a portion of the microsensor array; in this case, aportion of an array of microsensors with photosensitive area is, inaccordance with embodiments.

FIG. 3 depicts an exemplary implementation of a CMOS microsensor array,in accordance with embodiments.

FIG. 4 depicts a microsensor array, in accordance with embodiments.

FIG. 5 depicts a microlens cross-section, in accordance withembodiments.

FIG. 6 is a graph of transmittance (vertical axis) against wavelength(nm) (horizontal axis) depicting color filter and infrared rejection andadmittance, in accordance with embodiments.

FIG. 7 depicts an image processing system, in accordance withembodiments.

FIG. 8 depicts an image processing system, in accordance withembodiments.

FIG. 9 depicts a four pixel arrangement, in accordance with embodiments.

FIG. 10 depicts an image processing system, in accordance withembodiments.

FIG. 11 depicts a layout of color filters, in accordance withembodiments.

FIG. 12 depicts an interlaced row of visible light-sensitive andinfrared-sensitive pixels, in accordance with embodiments.

FIG. 13 depicts an interlaced row of visible light-sensitive andinfrared-sensitive pixels according to the present disclosure and beforeand after movement by the MEMS, in accordance with embodiments.

FIG. 14 depicts a method of writing a sensor array data in memory, inaccordance with embodiments.

FIG. 15 depicts a method of writing a sensor array data in memory, inaccordance with embodiments.

FIG. 16 depicts a method of writing a sensor array data in memory, inaccordance with embodiments.

FIG. 17 depicts a light filtering gate, in accordance with embodiments.

FIG. 18 depicts light filtering gates, in accordance with embodiments.

FIG. 19 depicts the image processing system, in accordance withembodiments.

DETAILED DESCRIPTION OF THE INVENTION

In some embodiments, the apparatuses, devices, and systems describedherein include: (a) a sensor array operable to sense a first type ofdata and a second type of data, the first and second types of data beingdifferent; and (b) a micro-electrical mechanical system (MEMS) to moveat least one of (i) the sensor array such that, at a common location,different sensors of the sensor array capture different ones of thefirst and second type of data and (ii) a filter positioned over at leastone of the sensors of the sensor array such that a selected sensor cancapture first or second type of data depending on the position of thefilter. In some embodiments, in a first operating mode, a firstcomposite frame are captured by the sensor array and, in a secondoperating mode, a second composite frame can be captured by the sensorarray. Each of the first and the second composite frames comprises firstand second types of data. In some embodiments, the first and secondcomposite frames can be divided into a first frame comprising only thefirst type of data and a second frame comprising only the second type ofdata. In further embodiments, the first and second frames are stored indifferent and discrete computer readable media. Alternatively, the firstand second frames are stored in different and nonoverlapping memorylocations.

In some embodiments, the first type of data is visible light (e.g., oneor more of red, blue and green light), and the second type of data canbe infrared light. The sensor array includes a first set of sensors tosense the first type of data and a second set of sensors to sense thesecond type of data. The first set of sensors preferably has differentmembership than the second set of sensors. In some embodiments, the MEMSmoves the sensor array from first to second positions to collect thefirst and second frames, respectively. The first and second sets ofsensors can be interlaced with one another on a row-by-row and/orcolumn-by-column basis. A distance and direction of movement of thesensor array by the MEMS is commonly a function of the type ofinterlacing employed.

In some embodiments, the MEMS move the filter positioned over at leastone of the sensors such that a selected sensor can capture first orsecond type of data depending on the position of the filter. The firsttype of data is commonly one or more of blue, green and red light andthe second type of data is commonly infrared light. In some embodiments,the filter block substantially (i) the one or more of blue, green andred light while passing infrared light or (ii) infrared light whilepassing the one or more of the blue, green and red light. In a firstoperating mode, the MEMS positions the filter over the selected sensorto filter light before the light contacts the selected sensor and, in asecond operating mode, the MEMS removes the filter from the selectedsensor to enable unfiltered light to contact the selected sensor.

In some embodiments, multiple filters are used. A first filter blockssubstantially the one or more of the blue, green and red light whilepassing infrared light and a second filter can block substantiallyinfrared light while passing the one or more of the blue, green and redlight. In a first operating mode, the MEMS positions the first filter,but not the second filter, over the selected sensor to filter lightbefore the light contacts the selected sensor and, in a second operatingmode, the MEMS positions the second filter, but not the first filter,over the selected sensor to filter light before the light contacts theselected sensor.

In some embodiments, the microsensor array is interfaced to a typicaldigital system, that reads and stores the data into a digital storage(memory) for subsequent processing, the whole operation being under thecontrol of a microprocessor (a CPU).

The present disclosure can provide a number of advantages depending onthe particular aspect, embodiment, and/or configuration. The imageprocessing system can be applied to make a sensor array sensitive withits full resolution of microsensor to more than one type of data. Fullresolution means that every pixel is used in the measurement of theincoming data.

CERTAIN DEFINITIONS

The phrases “at least one”, “one or more”, and “and/or” are open-endedexpressions that are both conjunctive and disjunctive in operation. Forexample, each of the expressions “at least one of A, B and C”, “at leastone of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B,or C” and “A, B, and/or C” means A alone, B alone, C alone, A and Btogether, A and C together, B and C together, or A, B and C together.

The term “a” or “an” entity refers to one or more of that entity. Assuch, the terms “a” (or “an”), “one or more” and “at least one” can beused interchangeably herein. It is also to be noted that the terms“comprising”, “including”, and “having” can be used interchangeably.

The term “automatic” and variations thereof, as used herein, refers toany process or operation done without material human input when theprocess or operation is performed. However, a process or operation canbe automatic, even though performance of the process or operation usesmaterial or immaterial human input, if the input is received beforeperformance of the process or operation. Human input is deemed to bematerial if such input influences how the process or operation will beperformed. Human input that consents to the performance of the processor operation is not deemed to be “material”.

The term “computer-readable medium” as used herein refers to any storageand/or transmission medium that participate in providing instructions toa processor for execution. Such a medium is commonly tangible andnon-transient and can take many forms, including but not limited to,non-volatile media, volatile media, and transmission media and includeswithout limitation random access memory (“RAM”), read only memory(“ROM”), and the like. Non-volatile media includes, for example, NVRAM,or magnetic or optical disks. Volatile media includes dynamic memory,such as main memory. Common forms of computer-readable media include,for example, a floppy disk (including without limitation a Bernoullicartridge, ZIP drive, and JAZ drive), a flexible disk, hard disk,magnetic tape or cassettes, or any other magnetic medium,magneto-optical medium, a digital video disk (such as CD-ROM), any otheroptical medium, punch cards, paper tape, any other physical medium withpatterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, a solidstate medium like a memory card, any other memory chip or cartridge, acarrier wave as described hereinafter, or any other medium from which acomputer can read. A digital file attachment to e-mail or otherself-contained information archive or set of archives is considered adistribution medium equivalent to a tangible storage medium. When thecomputer-readable media is configured as a database, it is to beunderstood that the database may be any type of database, such asrelational, hierarchical, object-oriented, and/or the like. Accordingly,the disclosure is considered to include a tangible storage medium ordistribution medium and prior art-recognized equivalents and successormedia, in which the software implementations of the present disclosureare stored. Computer-readable storage medium commonly excludes transientstorage media, particularly electrical, magnetic, electromagnetic,optical, magneto-optical signals.

The terms “determine”, “calculate” and “compute,” and variationsthereof, as used herein, are used interchangeably and include any typeof methodology, process, mathematical operation or technique.

The term “means” as used herein shall be given its broadest possibleinterpretation in accordance with 35 U.S.C., Section 112, Paragraph 6.Accordingly, a claim incorporating the term “means” shall cover allstructures, materials, or acts set forth herein, and all of theequivalents thereof. Further, the structures, materials or acts and theequivalents thereof shall include all those described in the summary ofthe invention, brief description of the drawings, detailed description,abstract, and claims themselves.

The term “module” as used herein refers to any known or later developedhardware, software, firmware, artificial intelligence, fuzzy logic, orcombination of hardware and software that is capable of performing thefunctionality associated with that element.

The preceding is a simplified summary of the disclosure to provide anunderstanding of some aspects of the disclosure. This summary is neitheran extensive nor exhaustive overview of the disclosure and its variousaspects, embodiments, and/or configurations. It is intended neither toidentify key or critical elements of the disclosure nor to delineate thescope of the disclosure but to present selected concepts of thedisclosure in a simplified form as an introduction to the more detaileddescription presented below. As will be appreciated, other aspects,embodiments, and/or configurations of the disclosure are possibleutilizing, alone or in combination, one or more of the features setforth above or described in detail below. Also, while the disclosure ispresented in terms of exemplary embodiments, it should be appreciatedthat individual aspects of the disclosure can be separately claimed.

Image Processing System

In some embodiments, the apparatuses, devices, and systems describedherein include an image processing system, or use of the same. The imageprocessing system comprises a sensor array able to measure more than onetype of measurement. This capability is realized by many differenttechniques. In various embodiments, the sensor array contains more thanone type of microsensor coupled with a micro-electromechanical system(MEMS) that allows this sensor to acquire the data as if it is formed ofa single type of microsensors. In some embodiments, the MEMS moves theentire sensor array; in other embodiments, the MEMS moves one or moreelements of the sensor array. The elements include one or more selectedsensors and/or one or more light filters positioned above one or moreselected sensors (or above the entire array). The movement of themechanical parts of this image processing system is generally a functionof the layout of the different types of microsensors in the sensorarray. As used herein, a sensor and microsensor are usedinterchangeably.

With reference to FIG. 1, a multiplicity or array of microsensors isarrayed in the form of a rectangular layout, though any geometricalpattern may be used. A first set of microsensors (labeled as 1 and 2) issensitive to a certain type of data, e.g., if the data are optical, thenit is sensitive to light in a certain wavelength, while the other secondset of microsensors is sensitive to another type of data, or anotherband of light. This embodiment does not depend on what type of data themultiplicity of microsensors is measuring. The concept can be applied toany array of microsensors, that traditionally was measuring one type ofdata. By adding to the array another type of microsensors, themicrosensor array can measure another type of data. As will beappreciated, more than two types of microsensors can be added to anyarray to measure more than two different types of data.

With continuing reference to FIG. 1, the array has a certain number ofmicrosensors along its width, W microsensors, and a certain number ofmicrosensors along its height, H microsensors, and the microsensors areinterlaced at a pitch of two in both the horizontal and verticaldirections (as shown in FIG. 1, where it is labeled “horizontal pitch”and “vertical pitch”). These W and H dimensions and pitch define thedirection and the magnitude of motion that the micro-electricalmechanical system must exert in order to make the microsensor arraymeasure both types of measurements without sacrificing the resolution ofthe measurement of either. In other words, the number of microsensorsmeasuring each type of data are the whole number of microsensors (WxH).

The Array of Microsensors

A general description of the array is depicted in FIG. 2. It shows theconcept of the microsensors, and their arrangement in the form of atwo-dimensional array. FIG. 2, shows a portion of that array which isformed of first, second, third, and fourth microsensors. Each of first,second, third, and fourth microsensors has a specific height and widthas defined by grid lines. A portion of this area is the specific portionwhich converts light to electricity; this area is labeled 4 a and 4 b.The ratio between this area and the area of the microsensor (which isthe width times the height), is called the fill factor of themicrosensor (the pixel fill factor). Therefore, when the light falls oneach of these microsensors, the light is converted to an electricalsignal that is read-out as an analog value. The choice of themicrosensor whose value is to be read out is done by choosing the row,labeled 5, and the column, labeled 6, that circuitry of this microsensoris connected to. This value is amplified via an amplifier, converted toa digital value, and stored in a memory for later use in a digitalcomputing and processing system (not shown).

The teachings of this disclosure apply equally to the sensor arrayconfigured as a CCD or CMOS and hence, the design of this circuitryconstitutes no bearing on the embodiments described in this disclosure.As will be appreciated, the two main approaches in manufacturinglight-sensitive sensor arrays are CCD (Charged-Coupled Devices) or CMOS(Complementary Metal Oxide Silicon). The access to the pixel isdifferent in both cases. In the case of CCD, the resulting voltageresults from a charge that is stored in the circuit elements of themicrosensor and is fully “coupled” to the neighboring pixel whose chargein turn is moved to its neighboring pixel, and so on until the whole rowis read out; hence, the name Charged-Coupled Devices (CCD). In CCD thepixels are readout sequentially and cannot be read at a random order. Incontrast, in CMOS any pixel can be accessed at random by choosing itsrow and column and reading the voltage value of that pixel.

In case of CMOS sensors, an embodiment of the circuitry that constituteseach one of these microsensors is depicted in FIG. 3. Where the lightfalls on the photosensitive element, it is responsible for convertingthe light signal falling on the microsensor into an electrical voltage,the size of this element as it is being laid out in the circuitry iswhat defines the fill factor of the pixel. The value of this voltage isread-out, if the ROW and COLUMN (COL) of this particular microsensor(pixel) are selected. The ROW and COL are selected via externalcircuitry called row and column decoders (which are building blocks ofthe whole array sensor). The mechanism of reading out of the value ofthe electrical charge depends on the sensor being CCD or CMOS. Theread-out mechanism has no implication or limitation on the embodimentsof the disclosure.

Because CMOS sensors are replacing CCD's in the field of imaging,read-out will now be explained for CMOS sensors. The electrical chargeis read out of this specific type of microsensor when the ROW (10) andCOL line (11) are activated by the external digital circuitry thatoperates the array. The external digital circuitry are called the rowand column decoders. The activation of the specific ROW and COL of amicrosensor allows the charge collected by the diode to be read-out viathe element Msel (9). The microsensor is reset in preparation for thenext measurement via the RST signal (12).

There are assistive tools that assist microsensors, specially opticalones, to gather optical (light) data. Examples of such assistive toolsinclude: the lens assembly, microlenses, and optical filters which blockcertain types of wavelengths, and those which admit certain types ofwavelengths. These filters can play an important role to make amicrosensor able to measure one type of data or another.

Example of Lens Assembly

For maximum optical coupling, i.e. to collect maximum amount of opticaldata, the pixel array can be covered with a lens assembly as depicted inFIG. 4. The mechanical components of the lens assembly include anoptical lens. The lens is housed, usually screwed in (to be used forfocusing the picture), in a barrel, that in turn covers the pixel array.The light that falls on the pixel array generally comes only through thelens. The center of the lens can be at the center of the pixel array. Aswill be appreciated, this cannot be guaranteed to be perfect, hence itis one of the parameters of a camera calibration process that iswell-known in the field of computer vision.

Microlenses

A microlens, as the name implies, is a structure that is put on everymicrosensor (pixel). It is a lens that typically covers one pixel only.Microlenses can increase the amount of light gathered on each pixelitself, hence increasing the ability of the corresponding sensors togather more light and therefore work at low light environment. Hence,microlenses can increase the signal-to-noise ratio of each pixel. FIG.5, shows a cross-section of a microlens. The light falls on a convexsurface of the microlens. The convex surface collimates the light tofall onto the photo-sensitive area of the microsensor (pixel), which iselectrically a photo-diode as explained above. The supportiveelectronics to collect the electrical signal resulting from the light isalso shown. The fill factor is also depicted as the width of the areasensitive to light to the width of the whole pixel. Additionally, inorder for the light not to leak from one pixel to another, solid sidesare built around the pixel, and is depicted.

Microsensors and Collecting Different Types of Data

To make a microsensor (a pixel) sensitive to one type of data whileblocking the other type of data, optical filters are employed to admitthe data that is required or desired while blocking the data that is notrequired or desired. By way of example, when the required or desireddata are visible image data (as type 1 data), and the undesired data areinfrared (IR) data (as type 2 data), the filters are used whoseresponses are depicted in FIG. 6. FIG. 6 shows the transmittanceresponses of filters that are applied to a selected pixel to receive andsense type 1 data while rejecting (or failing to sense) type 2 data orvice versa.

To receive type 1 data and reject type 2 data, the filters used to makethe microsensor able to receive light in the Blue, Green, or Redwavelength range, while rejecting the light in the Infrared range, haveadmittance responses (for the set of filters to admit Blue, Green, andRed, respectively). The filter needed for rejecting the infrared lighthas admittance response looking like the curve. It is worth noting thatthe IR filter can be a “coating” on the microlens itself.

By contrast to receive type 2 data and reject type 1 data, the filtersused to make the microsensor are able to receive infrared light only,while rejecting the visible light wavelengths, admit the infrared lightand conceptually looks like the filter with the admittance response.

Embodiments Using the Field of Imaging and Computer Vision

In the embodiments of imaging and computer vision, there are typicallytwo different types of measurements that need to be measured; hence,there are normally two types of microsensors (pixels) that are needed tomeasure these types of data. One type of microsensor is made sensitiveto visible light, and the other type of microsensor is sensitive toinfrared (IR) wavelength. This is particularly beneficial in the contextof developing a sensor array to be used in cameras that acquire imagesas well as depth data. However, the methodology explained here can beapplied to make a sensor array sensitive with its full resolution ofmicrosensor to more than one type of data. Full resolution means thatevery pixel is used in the measurement of the incoming data. Twodifferent types of data are selected to be measured to be related to thefield of imaging and computer vision. And these two types of data arevisible light intensity and infrared light intensity or, for short,video data and infrared data.

Imaging has traditionally acquired images and presented the images withthe highest quality to the users. Digital cameras replaced film due to aplethora of reasons, mainly ease of use, looking at the image beforetaking it, ease of erasing and re-snapping the image, and, above all,having the image in a digital format to be directly used by a computersystem that employs the images and/or video clips in different softwareprograms that further add value to these images.

Computer vision has become a field that is integral to different imagingapplications running on several devices that are used daily. Computervision goes to the next step after imaging, which is using theinformation in the image to achieve different tasks defined by the user.For example, computer vision applications find faces (e.g., facialrecognition) in the images and finds task-relevant objects to modifiedor augmented (such as in augmented reality). A prime example of thesesystems are the Human Natural User Interfaces (HNUI). These interfacesare used in many devices that are becoming more and more adopted byusers.

Depth measurement is often vital to computer vision systems and has infact been the holy grail of computer vision research. Many tasks ofcomputer vision required depth information of the scene that the camerais aimed at. Multiple-camera systems, mainly stereo vision(Stereo-vision is when two cameras are being used) has taken the lionshare of this research, but it had some drawbacks, system complexity byhaving two cameras and the more difficult ones are the correspondenceproblem and the occlusion problem. The first one is the problem ofmatching a region or a point in one camera image to the same point inthe other camera image. This problem becomes increasingly complex withmore than two cameras. Occlusion is the problem of not having the otherpoint in the other camera to begin with, because from the angle of theother camera it is being occluded by another object in the scene.

Confronted by these obstacles in relying on stereo computer visionalgorithms to find depth, other methods have emerged employingprojecting a priori known light patterns (visible or invisible to thehuman visual system) on the scene, and taking images of the scene toanalyze the deformation and size, among other properties of thesepatterns being imaged by cameras, to extract information about the depthof every point of the scene. It is better to use invisible patterns tocompute the depth so that they do not appear in the traditional images.For if the system were to be used for industrial applications, it wouldnot matter for the pattern to be visible or not, because these systemsare mainly concerned with quality control of the produced products. Butif the system were to be used in systems acquiring traditional picturesor video, it is generally not acceptable to see light patterns on thescene. The most commonly used wavelength spectrum for these systems isthe infrared wavelength spectrum. The camera that grabs regular videowill be referred to as a video camera, or video sensor, or video imager,while the camera that grabs the infrared spectrum will be referred to asthe infrared camera, or infrared sensor, or infrared imager.

The Image Processing System for Sensing Data

To create an image processing system that measures more than one type ofmeasurement, a separate sensor is employed for every type ofmeasurement. For example, if one captures visible light (digital camerasbeing prime examples) then there is a single sensor array that isretrofitted with the appropriate filters to capture just visible light(usually in the color bands Red, Green, and Blue).

FIG. 7 shows a system that captures visible light to present it in atwo-dimensional array. The basic idea of such systems, is to “sense” theenvironment using the two-dimensional array of microsensors under thecontrol of a microprocessor, or central processing unit (CPU). The CPUtypically programs different parameters on the sensor to make it moresuitable to sense the data and also drives a timing circuitry, whichtriggers the two-dimensional array to snap a two dimensional array ofdata using the array of microsensors. The CPU then drives the timingcircuitry to put the sensor array into the mode of reading out thedimensional data that it “sensed”. This data are stored into storagememory for later usage by any application that would utilize the senseddata. The first sensory data are the visible data collected by themicrosensor (pixels) of a camera. The storage memory is also referencedas video frame storage. Digital data are transferred from acrossdifferent system components: namely the sensor array, themicroprocessor, and the data storage elements (memory) via signalconductors which is also called in digital systems the data bus. Thedata are read out of the sensor array on the data bus via the signalcarrier, which is the wiring that carries the digital data coming out ofthe sensor array. This is the basic functionality of a digital systemdesigned to sense a single type of data. Obviously digital cameras havefar more features and many controls to the sensor itself, but they canbe readily implemented in the embodiments disclosed herein.

To expand the digital system to measure another type of data, anothersensor array is employed. This is shown using FIG. 8. For purposes ofillustration, the other type of data are infrared data to be used tocompute depth information of the scene. FIG. 8 contains a digital systemthat operates in a similar as the one depicted in FIG. 7 with theexception that the CPU send commands via the signal conductor to thetiming circuitry (which is identified as chip control logic). The timingcircuitry triggers the sensing of a second type of information using asecond sensor array besides sensing the first sensor array, and storingit in different memory storage (shown as depth frame storage) so as notto overwrite the first type which is stored in memory storage (shown asvideo frame storage). The first data are read out from the first sensorarray via the signal carrier, and the second data are read out from thesecond sensor array via signal carrier. The first and second data aretransferred to the appropriate memory via the signal carrier. Theexample of the second data is infrared data of the scene which the twodimensional sensor array is looking at. An additional optical lens canbe employed with infrared light signals to focus them on thetwo-dimensional microsensor (pixel) array (or second array) that isadded to sense the infrared data. In essence, the separate arrays anddefine two cameras with two lenses in one assembly (which greatlyreduces the compactness of the digital system).

Three-Dimensional Image Sensors and Methods of Manufacturing the Same

In some embodiments, there is a depth pixel as the fourth one of thesquare formed by that pixel itself and other 3 pixels that are sensitiveto Red, Green, and Blue colors, respectively, as shown in FIG. 9. TheFig. shows three pixels labeled R, G, and B, which are sensitive to thecorresponding colors of red, green and blue, respectively, while thepixel labeled D is the pixel that measures the infrared signal fromwhich the depth data will be later mathematically computed.

This approach will suffer the following drawbacks: the resolution ofboth measurements will lose resolution since the depth measurementpixels will remove some of the visible light sensitive pixel, the R, G,and B. Because the disclosed embodiment is a modification of the BayerPattern layout, the Red pixel is positioned under the Blue pixel whilethe Green pixel is removed from the second line. Because the Greenpixels contribute most to computing the intensity Y (in the domainYCrCb, Intensity, Red Chroma, Blue Chroma, respectively), and alsointensity is what defines the resolution of the picture (as opposed tothe Chroma), this type of approach will suffer a drastic reduction inits resolution, which is the most important aspect of imaging: namelythe sharpness of the image. This is such a damaging drawback, also thedepth resolution will be poor to begin with because it will be one pixelevery four (as seen in FIG. 9), i.e., one-quarter of the sensorresolution. In addition to those drawbacks, another defect is that thislayout will damage the Bayer pattern layout. This necessitates thedevelopment of completely new routines to reconstruct the video imagefrom the sensor array. The Bayer Pattern has become the standard forvideo processing from R, G, and B pixel arrays. Hence keeping it, allowsthe sensor to be readily used in almost any available image processingsystem. By contrast, the sensor assembly of the present disclosure canuse substantially the full resolution of the sensor array and eachsensor to measure both the depth, as well as the R, G, B signals. Thesensor array and digital system of the present disclosure can alsomaintain the Bayer-Pattern layout, which makes the sensor arraycompatible with the image processing systems in the market. Most of thepixel processing hardware processes images out of CCD or CMOS BayerPattern imagers.

Systems that Offer Only a Depth Sensor

Systems that use only a depth sensor are, as noted, flawed. They do notoffer video data and are merely “range” cameras. They typically includea light source, a detector, and a signal processor. The light sourcetransmits a source signal to the target according to a transmit controlsignal having reference time points. The detector receives a reflectedsignal from the source signal being reflected from the target. Thesignal processor generates a plurality of sensed values by measuringrespective portions of the reflected signal during respective timeperiods with different time delays from the reference time points. Thesignal processor determines a respective delay time for amaximum/minimum of the sensed values for determining the distance of thetarget. While the system of the present disclosure can be used as arange camera, it can perform a wide variety of imaging operations,including providing video data.

Three Dimensional Image Sensor

In some embodiments, the subject matter disclosed herein, the sensorarray and imaging processing system of the present disclosure canachieve the visible light and the infrared measure using a single array,without sacrificing the resolution at which each is measured. This canenable the array to be positioned be on a single or common chip that canoffer the major advantages of:

-   -   1—Compactness, hence amenable to miniaturization of the system        they are employed in.    -   2—Ease of system integration into other larger systems    -   3—Solves the problems of correspondence and occlusion which        arise from measuring with two different sensors.    -   4—Accurate depth and video data matching due to the physical        maximum proximity between pixels (in embodiment 1) and the use        of the same pixels for both, image and depth (in embodiment 2)        due to the measuring of the data by the two types of pixels        being at the same location, in embodiment 1 by moving the sensor        pixel array, and in embodiment 2 by moving the filters that        cover the pixels. Hence virtually there is no difference between        in the localization of the image and depth sensing data.    -   5—One set of intrinsic camera parameters due to the use of one        array, and one lens.        Description of the Image Processing System of the Present        Disclosure

This description can be divided into two main parts:

-   -   A—The microsensor array design and the different approaches used    -   B—The digital circuitry        The Microsensor (Pixel) Array and the Different Approaches to        the Layout Design of the Array:

The microsensor or pixel array is the heart of the sensor that, coupledwith the microelectromechanical system (MEMS), measures multiple datatypes. Although the system is discussed with reference to only two datatypes, the system can be easily expanded to measure more data types. Theexemplary two data types used in this discussion are video data as wellas IR data. The latter type of data can, using mathematical algorithms,be converted to depth data.

There are two main philosophies in achieving the system pixel array withthe surrounding supporting circuitry that can deliver multitype sensorydata:

The first approach uses geometrical interlacing arrangements betweentype 1 microsensors; namely, image visible light pixels and type 2microsensors (e.g., the infrared pixels). This is known asspace-multiplexing, where the image sensor real-estate is shared betweentype 1 microsensors (image pixels sensitive to visible light) and type 2microsensors (infrared light sensitive pixels). In this case and asdiscussed below, the micro-electrical mechanical system (MEMS) will movethe whole sensor array in a direction and with a magnitude that is afunction of the geometric interlacing pattern.

The data output in this approach is a mix of type 1 data and type 2data.

The second approach uses the whole sensor array to be a type 1 data(image) sensor and also the whole array as type 2 data (IR/depth)sensor. This is frame or time multiplexing. In this approach, MEMS areused to cover each sensor of the array with filters appropriate for themeasurement. For type 1 data, the MEMS move a first filter stack on themicrosensor that allows type 1 data to be measured and block type 2 datafrom measurement. In the other mode, the MEMS allows another secondfilter set to be on top of the microsensors that allows type 2 data tobe measured. The second filter set may block from measurement all orpart of the visible light spectrum. There are two different embodimentsto achieve this approach.

The data output can be either all type 1 or all type 2 data or both type1 and type 2 data. The whole array in a frame acts as an infraredreceptor, and in another frame as a visible light traditional visiblelight camera sensor. Thus, in a first frame, infrared light is collectedand, in a second frame, visible light is collected, or vice versa.

The details of embodying these two design philosophies will be explainedlater.

In the following section, an embodiment of a digital system is depictedin which the new pixel array system comprises the components of asingle-chip sensor.

The Image Processing System

The previous approaches to the sensor array are embodied in a digitalsystem, which has also other supportive circuitry components to achievethe implementation of the approaches as well as provide thefunctionality needed to make the sensor able to deliver video and depthdata. An example embodiment of the pixel array and supportive circuitryis explained using FIG. 10. The lens is positioned above the imageprocessing system and gathers and directs the optical signal to thewhole array. The system represents the components that would be on acommon chip or circuit board.

The pixel array is where a multiplicity of microsensors are arranged toachieve the measuring of more than one type of data. There is more thanone approach to design the layout of the microsensors. These approacheswill be explained later.

The read-out is performed differently for CCD and CMOS technologies.When the sensor is a CCD technology the read-out decoding logic drivesrow and column circuitry that causes each row to transfer its chargefrom one pixel to the next till the whole row is read out. Hence thisread-out decoding login manages the sequential read-out of the arrayvalues. By contrast when the sensor is CMOS technology, the row andcolumn decoders rather than read-out logic, select the pixel(s) whosedata are to be read out, which read out can be sequential or random. Theread-out sequence and the storage of the data in memory in done in a waythat maintains the order of the data will be explained later inconnection with the implementation of the design approaches mentionedabove. A correlated double sampling (CDS approach is commonly used inreading data out in CMOS sensors).

The analog gain circuitry amplifies values as the analog value of eachpixel is read-out.

The analog to digital converter converts the pixel analog data into adigital value, and the digital gains circuitry applies a multiplicationfactor to the pixel data that is read-out. This module can be present onthe chip or implemented in other parts of the system. The digital datacan be stored in a register prior to reading it out to the rest of thesystem via the digital bus.

The Micro-electro Mechanical System (MEMS) includes MEMScontroller/circuitry in signal communication with the actuators to movethe array. The MEMS can be an important component of the chip. The MEMSmoves different parts of the chip to achieve a goal of the imageprocessing system, which is measuring more than one type of data with asingle sensor, both at the full spatial resolution of the sensor.Depending on which design approach is implanted in the sensor array, theMEMS will move different sets of parts. MEMS controller/circuitryprovides the control to the electromechanical actuators. The actuatorsare commonly divided into two blocks logically because physically theymay be one block on the chip.

The chip control logic receives data from the CPU (not shown) to programhow the whole image processing system and its component parts (includingthe MEMS and sensor array) will operate, via the module Array OperationLogic. Examples of such parameters are: the exposure time of the sensorarray to gather a selected type of data (which may vary by type of datato be collected), the gains applied by the analog and/or digital gaincircuitry to amplify the collected sensor data, the speed of clockingthe sensor array (which may vary by type of data to be collected), andso on. Other important information can include: the timing to drive theMEMS to move the actuators to synchronize with grabbing or collectingthe data and the timing and type of IR light pattern to project on thescene. The MEMS motion of the sensor is explained below in connectionwith the different approaches of designing the sensor array or imageprocessing system. Also the concept of using IR light patterns torecover depth information will be explained below.

The digital storage and its access can be any type of computer readablemedium located on or off the chip or both. This is a system design issueand does not affect the functionality of the sensor array, provided thatthe design is done with major consideration to the speed of reading outthe data from the sensor array. Storing the data in the memory can bedone by the microprocessor (not shown), which would read out everydigital data of every pixel and store it into a digital storage, or byhaving a “Direct Memory Access” (DMA) digital block that off-loads thecollected digital data from the register.

The wiring (the bus), that carries the sensed information or data in andout of the chip, transfers the data from the CPU (not shown) to theinternal digital blocks (not shown) of the image processing system,which controls the image processing system's different operations. Thewiring also carries the data being read out from the sensor array, viathe digital output lines labeled, to the appropriate digital storageblock (digital memories).

Other components play a role in receiving depth-related data from thesensor array, which is used to compute the depth data of the scene orimage. These components are infrared (imperceptible) light patternemitters that will project these patterns outwards onto the scene orimage. For achieving maximum system integration, the light-patternemitter(s) can be integrated on the same chip.

There proposed embodiments of the two approaches are discussed below inmore detail.

The First Approach of Implementation The First Embodiment

The image processing system includes a (micro)sensor array where thereis a mix between type 1 and type 2 microsensors, e.g., between regularvisible color-selective pixels (sensitive to Red, Green, and Blue) aswell as infra-red sensitive pixels. The pixels are located on a commonsubstrate, preferably in an interlacing pattern. The interlacing patterncan be in the rows or in the columns, or in both.

An embodiment of such an arrangement is seen using FIGS. 11 and 12.First, in FIG. 11 a layout of microsensors (pixels) is depicted that aresensitive to type 1 data only, the visible light, where the microsensorsare covered with color filters to make each pixel sensitive to one typeof color only. The sensors are located on a common chip or substrate. Ina selected portion of the array, first and second sensors and aresensitive to GREEN light, a third sensor to BLUE light, and a fourthsensor to RED light. The shown pattern is known as Bayer pattern, whichis the most widely used pattern in color imaging sensors.

FIG. 12 shows how the sensor array shown in FIG. 11 is converted tosense more than just the visible light. This is done by interlacing thepattern of microsensors that are sensitive to type 1 data (the visiblelight BLUE, RED, and GREEN sensitive pixels) with microsensors that aresensitive to type 2 data, namely the infrared light (IR), where theinterlacing is done in the rows. Thus, the rows represent the Bayerpattern while intervening rows are sensors sensitive to type 2 data.Thus, there are two types of interlacing, a first is interlacing betweenBLUE, GREEN and RED light sensitive pixels and a second is interlacingbetween BLUE, GREEN and RED light sensitive pixels on the one hand andtype 2 data, or infrared, sensitive pixels on the other.

The manufacturing of the sensor array follows the common way ofmanufacturing CMOS cameras, hence it will be straightforward for currentCMOS manufacturers to produce such a sensor. This manufacturing processincludes the following steps:

-   -   (1) All microsensors (pixels) are manufactured in a similar        layout,    -   (2) A—Color filters (with the visible light filter admittance        response shown in the FIG. 6, labeled B, G, and R) are applied        to the pixels in FIG. 11 that will be used for visible light        sensitivity in RED, GREEN, and BLUE. In the current embodiment,        the pixels are applied to row 0 and row 1, as Green and Red        interlacing pixels on row 0 and Blue and Green interlacing        pixels on row 1, then skip row 2 and row 3, and then applied to        row 4 and row 5 similar to row 0 and row 1 respectively, and so        on till the end of the array.    -   (3) B—An IR blocking filter is applied, on those rows that will        be sensitive to visible light, namely: rows 0, 1, and row 4, 5,        . . . in the current embodiment, as they are shown in FIG. 12.    -   (4) The other rows, row 2 and row 3, and then row 6 and row 7,        and so on . . . will be the IR sensitive pixels, where the IR        blocking filter and the color filters are NOT applied. In FIG.        12.    -   (5) Microlenses increase the amount of light gathered on each        pixel itself. They are applied on each pixel of the whole pixel        array to increase the light-gathering efficiency of each pixel.

The sensor can be used in this fashion, sacrificing half of the verticalresolution of collecting both type 1 and type 2 of data. The presentdisclosure overcomes this problem by using MEMS to move the sensor up(the direction in which any resolution is lost, in this case, it is inthe vertical direction but can be in another direction depending on theapplication) by a distance equal to the length dimension of two pixels,along the same direction (in the case shown in FIG. 12 the distance inthe vertical dimension of the pixel (relative to the plane of the page)is labeled “h”). It is to be appreciated that the distance “h” is notlimited to the length dimension of two pixels but can be one or morethan two pixels depending on the application. The MEMS design isstraightforward since it can effect the motion of the array in only onedimension. As will be appreciated, the distance and direction of motionof the sensor array is a function of the interlacing pattern of thedifferent types pixels that gather different types of data. The presentembodiment shows interlacing only in the vertical direction (in theplane of the page); however, the present disclosure can be easilyextended to interlacing horizontally (in the plane of the page). In thatcase, the sensor array is moved horizontally (in the plane of the page).It can also use interlacing in both the vertical and horizontaldirections (relative to the plane of the page) in which case the sensorarray is moved in both the vertical and the horizontal directions(relative to the plane of the page).

Stated another way, at a first time and in a first operating mode all orpart of the pixel array is in a first position. In this position thepixel array reads both type 1 and type 2 data in the certain spatialposition as shown by the spatial layout of the microsensors. At adifferent second time and in a second operating mode, all or part thepixel array is moved by the MEMS to a second position different from thefirst position. In this position, the pixel array reads still both typesof data, type 1 and type 2. When the all or part of the sensor array ismoved up to position 2 so as to reverse the locations of collecting type1 and type 2 data, hence there is no loss in the spatial resolution ofboth type 1 and type 2 data measurements.

In another configuration, at a first time and in a first operating modeall or part of the pixel array is in a first position. In the firstposition, the type 1 data sensitive pixels are positioned to collecttype 1 data while the type 2 data sensitive pixels are positioned so asnot to collect type 2 data. In this position, the type 1 data are readout of the type 1 data sensitive pixels. In another embodiment, the type2 data sensitive pixels can collect type 2 data in the first operatingmode.

At a different second time and in a second operating mode, all or partof the pixel array is moved by the MEMS to a second position differentfrom the first position. In the second position, the type 2 datasensitive pixels are positioned to collect type 2 data while the type 1data sensitive pixels are positioned so as not to collect type 1 data.In this position, the type 2 data are read out of the type 2 datasensitive pixels. In another embodiment, the type 1 data sensitivepixels can collect type 1 data in the second operating mode.

This is demonstrated by FIG. 13, which also shows how there is no lossin the special resolution of the measurements. FIG. 13 shows the pixelarray depicted in FIG. 12 with the whole array before motion (the leftarray) (or when the array is in the first operating mode and in thefirst position) and after being moved up by a distance that is equal tothe interlacing amount (which is two rows) by a distance equal to thevertical height of two pixels so that the pixel array is in the secondoperating mode and in the second position.

From the interlacing pattern and the motion of the sensor it can beappreciated that the data taken from the two positions of the sensorwill complement one another to create a full resolution of type 1 ofdata video data as well as full resolution of type 2 of data infrareddata. This sequence of data acquisition can be further explained as:

Sequence of Operation of the Pixel Array in this Approach:

The operation of image acquisition in the current embodiment isexplained only in the case of pixel layout shown in FIG. 2, where theinterlacing is done in the vertical direction (in the plane of the page)but the operation is identical for the horizontal direction (in theplane of the page). The operation can be logically extended in any otherpattern of interlacing.

In the current implementation where the interlacing of image pixels andinfrared-sensitive is done along the vertical direction (in the plane ofthe page), the sequence of operation image acquisition of the sensor isoutlined as follows:

-   -   1—A frame (or frame A) is captured, and is read-out and stored        in memory (the storage operation is explained in the previous        section as is shown in FIG. 10).    -   2—The sensor array is moved up (in the plane of the page).    -   3—Another frame (or frame B) is captured, and is read-out and        stored in (the storage operation is explained in the previous        section as is shown in FIG. 10).    -   4—The data from both frames are used to generate a full frame of        type 1 data (visible image frame to be used in the video        processing), and a full frame of type 2 data (infrared frame to        be used in the depth calculation).    -   5—steps 1-4 keep repeating as long as the image processing        system is operated to gather these two types of data.

In the following section, the mechanism of the digital storage of themixed types of data that will be output from the sensor is explained.

Reading and Storing the Mixed Data

When the data are interlaced in this fashion, the type 1 and 2 data mustbe stored in memory in a different way, so that in the end, type 1 dataand type 2 data are stored contiguously and sequentially together as ifthe two types of data are coming from two different sensors and in thecorrect order of the rows. First, it is important to understand how thedata are sequentially read out and stored in case of a traditional arrayof microsensors, i.e., which measures a single type of data. FIG. 14shows the storage of a single type of data coming sequentially alongeach row from an array of a single type of microsensor. The data arestored digitally in a digital storage one row after the other (with eachrow stored as one pixel after the other usually from the left of thearray to the right). Row 0 is followed by row 1, then row 2 till row n,which is the last row to be read out from the array. So the data of thesame type is stored contiguously. In the present disclosure, it isdesirable to achieve the same outcome with the storage of more than onetype of sensed data. That is data of type 1 (video data) is stored alltogether and that of type 2 (IR data) is also stored all togethercontiguously. The method of achieving this will now be explained usingFIGS. 15 and 16.

Referring to FIG. 15, the data coming out of the microsensor array issequential; that is, the data are read out from the sensorpixel-by-pixel in one row, then pixel-by-pixel in the next row, then thenext, and so on until the end of the array. Hence, the data read outfrom the array is a mix of both type 1 and type 2 data. Namely, the readout data are two rows of video data, and two rows of IR data. Usingsimple digital logic circuitry, the data are directed to be stored indifferent and/or separate storages, one for type 1 and the other fortype 2; e.g., the type 1 data are stored in a video data storage areaand the other in an IR data storage area. This has to happen whilepreserving the order of the data of each type. This is achieved asdepicted in FIG. 15 as follows: the type 1 data of row 0 and row 1 isstored in the type 1 storage memory (the video memory as shown bytransactions and in FIG. 15). After storing rows 0 and 1, skip in memoryenough space to store row 2 and row 3 of the same type of data (e.g.,video data) because it is NOT there yet (it will be available after MEMSmoves the sensor array to the second position and grabs another frame ofmeasurement). Then, rows 4 and row 5 are stored in the type 1 memorystorage (as shown by transactions). But rows 2 and 3 are the type 2 ofdata; hence, the type 2 data are stored in the other storage memory (theIR memory) after skipping in memory enough space to store rows 1 and 2of the same type of data (e.g., IR data) because it is NOT there yet (itwill be available after MEMS moves the sensor array to the firstposition and grabs another frame of measurement). After storing tows 2and 3 (as shown by transactions), skip in memory enough space to storerows 4 and 5 of the same type of data (e.g., IR data) because it is NOTthere yet (it will be available after MEMS moves the sensor array to thefirst position and grabs another frame of measurement). Then, rows 6 and7 from the current frame are stored in type 2 memory storage, and so on.

When the sensor is moved to the second position, as shown in FIG. 17,and another set of data are collected, row 4 and row 5, which arephysically located after motion up by two rows in the location of row 2and 3, are now used to store the missing video data of row 2 and row 3,as shown by transactions, respectively. In this case, rows 0, 1, and 4and 5 are of type 2 of data, and they are stored in their appropriatelocation as shown, for the type 2 data of rows 0 and 1, by transactionsin FIG. 16. The same concept is applied to the whole sensor arrayvertically.

At the conclusion of this process, all type 1 data are storedcontiguously in one memory (e.g., color image row storage), while alltype 2 data, the full resolution of the sensor, are stored in the memory(e.g., IR image rows storage).

The pixel layout can be changed depending on the applications. Forexample, the interlacing can be done along the horizontal direction, inother words, there is one column of color-filter covered pixels, and thenext column is infrared-sensitive pixels. The sensor pixel array, inthis case is moved along the horizontal direction by an amount that isequal to the horizontal length of the pixel.

Along the same idea, different pixel patterns can be also implementedsacrificing the resolution of one type of measurement compared to theother (for example having high resolution of the video data versus thedepth data). For example, if it is preferred that the sensor to havebetter resolution is for the video data while the resolution of depthdata is sacrificed, two rows of the infrared-sensitive pixels can beused for every four rows of color-filter-covered pixels. In that event,the pixel array is moved along the vertical direction by two pixels,hence, the full Bayer Pattern is reconstructed while missing two rows ofthe vertical resolution of the infrared data. The same approach can beapplied by having the interlacing done in the horizontal direction. Aswill be appreciated, there are different combinations that the sensorand system architect can implement. But the teaching of the presentdisclosure remains the same: that is, the teaching is to interlace themicrosensors and move the sensor to covered the interlaced area(“interlace and move”).

In many embodiments, the movement and the stoppage of the array isguided by limit switches, labelled 400 and 401 in FIG. 10. They operateon the such that when the image array is moved, the end of its travel issensed when it causes these limit switches to change their state, fromexample from OFF to ON (or vice-verse), in order to signal that thearray has reached its final destination. This allows the actuatingcircuitry that moves the array to halt its operation. At this point thearray is stationary in its new position. Each of the limit switches iscoupled to corresponding circuitry in order to transmit signals from thelimit switch to the circuitry in order to store data of the sensor arrayaccording to the configuration of the sensor array.

Each of the limit switches is configured to transmit a signal to thecircuitry to indicate the change of state of the limit switch. When thesensor array arrives at a first position, the limit switch transmits afirst signal to the circuitry to indicate that the sensor array comprisea first configuration. When the sensor array arrives at a secondposition, the limit switch transmits a second signal to the circuitry toindicate that the sensor array comprises a second configuration.

In accordance with the embodiments of FIGS. 15 and 16, the array in itslower position comprises a first configuration of the sensor array, asthe sensor array is in contact with the lower limit switch (in FIG. 15,labelled 700). When the sensor array has moved to the upper position,sensor array comprises a second configuration, in which the movement ofthe sensor array has been limited upon contact and actuating the upperlimit switch (in FIG. 16 labelled 701). In many embodiments, the limitswitches provide a signal to the circuitry upon the sensor arrayreaching a desired location, which is marked by the position of theswitch. In accordance with many embodiments, the limit switch maycomprise more than two limit switches, for example with embodiments inwhich the array is stopped the array in more than two locations.

The Second Approach of Implementation The Second and Third Embodiments

Unlike the first approach, the second approach uses the same microsensorto measure different types of data; that is, the same pixel measuresboth visible and IR data. This can only achieved by preparing the pixelwith the appropriate filters that admit a certain type of data andblocks the other type(s), and then changing it to admit the anothertype, and blocking the first.

This is achieved using the MEMS to cover each pixel with the filtersthat convert back and forth from being an infrared sensitive pixel to avisible light pixel.

In this approach, each pixel itself is covered with a microlens, but notwith color filters, IR admittance, or IR blocking filter.

In the infrared or second operating mode the pixel is exposed toincoming light without the interference of any filter in its path(obviously except for the camera lens). In the visible light or firstoperating mode the MEMS actuates a gate that has on it the IR filter andthe appropriate color filter for every pixel (Red, Green, or Blue). InFIG. 15, this concept of the image sensor array is shown using thecross-section of one pixel.

Each pixel has a photo-electric area which is in charge of collectingincoming light and converting it to an electric signal that is afunction of the intensity of that light. A side or cross-sectional viewof this area is shown in FIG. 17. A cross-section of a light filtrationassembly or “gate” (a rectangular or a square structure), which carrieson it, the IR blocking filter and a visible light admittance filter (inone of Red, Green, or Blue depending on the pixel) can be rotated to agate open position to subject the pixel to incoming light or a gateclosed position to remove IR and certain visible light spectra from thelight impacting the pixel. In the gate opened position (shown on thebottom left), the pixel will be sensitive to infrared light, whereas inthe gate closed position (shown on the bottom right) the pixel will actas an image pixel that is sensitive to desired wavelength spectrum ofvisible light, which is one of Red, Green, or Blue, depending on thecolor filter used for this particular pixel. A layer of a transparentmaterial (can be glass) is laid on top of the door before putting the IRand the color filters. This layer is put to support both the IR and thecolor filter due to the fact that the light filtration assembly isopened from the middle so as to let the light through when it is in theclosed position (shown on the bottom right).

FIG. 18 depicts, a second embodiment, a more complicated variation ofthe gate approach, where two gates are used so that an IR admittancefilter is applied to the pixel when it is in the mode of acquiringinfrared light. In FIG. 18, there are two gates in the top center ofFIG. 18. When the camera is acquiring a video image (in the bottom rightof FIG. 18), gate is closed and another gate opened to take the infraredadmittance filter “out of the way” and have the color filters and theinfrared blocking filter in the way of the light hitting the pixel. Whenthe camera is put in the mode of acquiring an infrared image, then theopposite is done, namely the gate is rotated out of the way while thegate is rotated or closed to have the IR admittance filter in the way oflight hitting the pixel, as shown in the bottom right of FIG. 18.

Another, third variation, of this embodiment is making the gates largeenough to cover the whole pixel array as opposed to a pixel-by-pixelbasis. Where there can be two gates of the same concept as gates made ofglass, or they can be rectangular frames covered with a plane of glass(depending on feasibility of manufacturing), and they can close or openon the whole pixel array. Compared to a gate on every pixel, thissolution is simpler, but it may suffer from larger inertia required toopen and close the gates and will require vertical room above the pixelarray to accommodate the opening of the gate which is now as high as thepixel array and as long (or as wide) as the whole pixel array, dependingalong which side it opens and closes.

Both approaches can provide a full resolution Bayer Pattern of visiblelight R,G,B pixel array, as well as a full-resolution infrared sensitivepixel array. Therefore:

-   -   1—The Color Filter pattern is still preserved as a        Bayer-Pattern, hence all of the already available pixel        processors software and hardware will be compatible with this        image/depth sensor. This is a major advantage since the        Bayer-Pattern processing is extremely mature and is almost the        standard in image processing systems for CMOS (as well as CCD)        cameras. The current embodiments however can be applied to any        pattern, and also it is worth noting that the R,G,B light        admittance filters can be any other triplet of color filters        (such Cyan, Magenta, and Yellow, or others) as long as their        combination can provide all the color, including white.    -   2—In the first approach, where the whole sensor is moved, there        is limited deviation from the regular manufacturing of regular        CMOS video sensors. Hence, current technology is readily mature        to implement this first embodiment readily. Also the required        motion from the MEMS is straightforward.        Computing Depth from the Infrared Measurement

What the present disclosure means by infrared measurement is the use oflight, or rather structured light patterns, whose wavelength falls inthe infrared light spectrum. And the reason for that is to avoidinterference between these light patterns with other “perceptible” or“visual” imaging functionality, such as image acquisition, so that theselights do not “appear” in the acquired image. Structured light is theprocess of projecting a known pattern of pixels onto a scene. Theselight patterns can be in the form of shapes lines, grids, and/orcircles. The way that these deform when striking surfaces allows visionsystems to calculate the depth and surface information of the objects inthe scene. Or these shapes can be in the form of functions, such aslight intensity grading in the form of a sinusoidal function or a squarefunction, with different frequencies. The change of the “phase” asdetected by the sensor from that of the projected light is a function ofthe depth of that point in the scene. There are multiple publicationsthat cover these algorithms and different enhancements of it. In thepresent embodiment here, any of these algorithms can be employed.Humanly invisible (or imperceptible) structured light uses light thathas wavelengths that would not interfere

B—The Embodiment of a Digital Circuitry with the Pixel Array

The present disclosure has covered the different approaches fordesigning a single pixel array system-on-a-chip to achieve measuringmore than one type of data with the same array. Now an exampleembodiment of this “chip” will be shown. The present disclosure hasfurther explained the methodology of storing the data out of the sensorinto the memory when the data are a mix of the two types of sensed data.An embodiment of a digital system that implements the above is describedbelow.

The pixel array simultaneously measures more than one type of data(visible light which is used to form video image and infrared light thatis used to compute the depth measurement). The array can be implementedin any of the different embodiments explained above. The pixel arraywith the MEMS are encapsulated together on a single chip. The imageprocessing system operation is controlled by the processing unit calledthe Image Processing Unit. The Image Processing Unit puts the imageprocessing system in the first operating mode of composite type 1 data(e.g., the visible data mode) and type 2 data (e.g., the IR mode)sensing or the second operating mode (by moving the array) of thecomposite type 1 data and type 2 data sensing. Or in another embodiment,sensing type1 data in a certain mode, while sensing type 2 data inanother mode. The Image processing system mode is controlled by theCentral Processing Unit (CPU) in FIG. 7. The CPU also drives the TimingCircuitry in FIG. 10 (via the wiring bus in FIG. 7), which triggers thetiming to the two-dimensional array image sensor to “sense” the scene(snap the image). After that the timing circuitry drives the sensorarray to read-out the data gathered in the microsensors (pixels) overthe read-out bus from the sensor array. The data can be of type 1 and oftype 2 or of both together (depending on the approach used for thesensor design). The data of both types (pixel data) are then processedby the CPU and the memory to be gathered to be stored in theirappropriate corresponding storage of type 1 data (memory) and type 2data (memory).

The data are not stored in a mixed fashion, although in one of theembodiments discussed herein they are collected in a mixed fashion. Type1 data (e.g., video data) are stored in video frame storage area,whereas type 2 data (e.g., the IR data) are stored in depth framestorage area). Now, the image processing system can generate type 1data, a video image, or type 2 data, IR image (depending on theoperating mode in which the array is in), and the image is stored in acorresponding area (video memory for video mode, and depth memory fordepth mode), both to be used later in applications. The CPU can be alsoa pixel processor that achieves all the processing of video and IR. Incase of the video processing of the Bayer pattern, pixel interpolationis used by the CPU to compute color components RGB for each pixel. Otheroperations include, without limitation, digital gain, white balance,color space conversion (from RGB to YCrCb), and gamma correction. Incase of infrared data processing, the CPU performs, but not limited to,data noise removal and pattern deformation recognition and depth dataand/or object shape recovery. The CPU can use the IR image to calculatethe depth data (using well-known publicly available algorithms) andstore it in the depth frame storage or it can be passed to otherapplications that will run on the systems that will comprise theembodiments herein. The data transfer between modules is carried outusing digital bus(es). Bus is used to read data out of the sensor andbus to write data to the sensor. In digital logic terminology, a bus isa multiplicity of wires on which digital data get transferred in adigital system from one digital circuitry to another in the system.

To collect useful IR data that will be used to compute depth, the imageprocessing system projects one or more light patterns on the scene.Hence the image processing system can have one or more infrared lightemitters. The IR emitters can be placed all around the pixel array. Alsoeach emitter can project a different pattern, which is used in depthmeasurement. The light pattern is projected onto the scene, thereflection of the pattern from scene is received by the pixel array andwith different computing algorithms, depth data of the scene iscomputed. Preferably, the light pattern is not visible to the user;otherwise, the scene will be filled with “artificial” light patternsthat appear to the user, which would greatly interfere with thetraditional image snapping and grabbing that the user is familiar with.The scene should therefore be void from anything visible that interfereswith the scene, and hence will be recorded in the image that the user isgrabbing.

As mentioned, the MEMS makes the sensor array of the present disclosurecapable of collecting different types of data using the same microsensor(pixel) array. The image processing system achieves the measurement ofthose different types of data by moving certain parts of our sensorarray system. In the different embodiments, the MEMS moves one or morecomponents which can differ depending on the configuration of the imageprocessing system. The MEMS, however, is implemented in all of theembodiments to achieve the goal that the same sensor array can delivertwo different types of data, which has been described as a video image,and an IR image.

Other Sensory Components on the chip can be, but not limited to:

-   -   A—Infrared Emitters with different patterns, which are depicted        as the three circles on top of the sensor array in FIG. 10 (and        shown also in FIG. 17). There is virtually no limit on their        numbers and there is no limitation on which wavelength band and        distribution they work in as long as they are visible by the        infrared microsensors and invisible to the regular visible light        microsensors (the color image pixels).    -   B—A MEMS microphone array on the chip    -   C—X-Y-Z Accelerometers and and X-Y-Z gyros    -   D—A strobe light i.e. flash (in FIG. 19) to assist in improving        the quality during acquiring images        Applications:

Due to its high integration and the advantages of having a single chipsolution to imaging and depth capturing, the proposed image processingsystem on a chip can be used in, but not limited to:

-   -   1—Gaming; high integration by having a single chip that emits        the light and acts as both depth and video sensor, enables any        video camera in a system to be replaced with our invention. This        facilitates integration of 3D imaging applications on        practically all computing and communications devices.    -   2—Human user interfaces for computing and communication devices,        where it can replace all the cameras on laptops and mobile        devices and surface-interaction computing pads. This will give        rise to novel applications where there our device can be used as        an image sensor, or a depth sensor, or both.    -   3—Augmented Reality applications will be greatly facilitated to        be coded on mobile devices, tablets, and laptops due to the        presence of a 3D camera on the device. This is because 3D data,        video data are integrated into a single sensor. In Augmented        Reality applications these data (video and three-dimensional        data of the scene, which are already spatially congruent        together because they are coming from the same sensor), are        integrated with three-dimensional graphics that are suited for        different applications: video gaming, education, training, and        marketing)    -   4—Security/Surveillance where 3D tracking of perpetrators is        achieved, which can create novel algorithms.    -   5—3D printing, the current sensor can be interfaced via a        digital system that comprises hardware and software to print 3D        data which acquired by the sensor and processed to recover the        3D data from the measurements and send it to the printer.

The exemplary systems and methods of this disclosure have been describedin relation to microsensor arrays. However, to avoid unnecessarilyobscuring the present disclosure, the preceding description omits anumber of known structures and devices. This omission is not to beconstrued as a limitation of the scopes of the claims. Specific detailsare set forth to provide an understanding of the present disclosure. Itshould however be appreciated that the present disclosure may bepracticed in a variety of ways beyond the specific detail set forthherein.

Furthermore, while the exemplary aspects, embodiments, and/orconfigurations illustrated herein show the various components of thesystem collocated, certain components of the system can be locatedremotely, at distant portions of a distributed network, such as a LANand/or the Internet, or within a dedicated system. Thus, it should beappreciated, that the components of the system can be combined in to oneor more devices, such as a camera or other imaging device, or collocatedon a particular node of a distributed network, such as an analog and/ordigital telecommunications network, a packet-switch network, or acircuit-switched network. It will be appreciated from the precedingdescription, and for reasons of computational efficiency, that thecomponents of the system can be arranged at any location within adistributed network of components without affecting the operation of thesystem. For example, the various components can be located in a server,in one or more computational devices, at one or more users' premises, orsome combination thereof. Similarly, one or more functional portions ofthe system could be distributed between a telecommunications device(s)and an associated computing device.

Furthermore, it should be appreciated that the various links connectingthe elements can be wired or wireless links, or any combination thereof,or any other known or later developed element(s) that is capable ofsupplying and/or communicating data to and from the connected elements.These wired or wireless links can also be secure links and may becapable of communicating encrypted information. Transmission media usedas links, for example, can be any suitable carrier for electricalsignals, including coaxial cables, copper wire and fiber optics, and maytake the form of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Also, while the flowcharts have been discussed and illustrated inrelation to a particular sequence of events, it should be appreciatedthat changes, additions, and omissions to this sequence can occurwithout materially affecting the operation of the disclosed embodiments,configuration, and aspects.

A number of variations and modifications of the disclosure can be used.It would be possible to provide for some features of the disclosurewithout providing others.

For example in one alternative embodiment, the sensor array of thepresent disclosure can be used for patterns other than the Bayerpattern.

In another alternative embodiment, the sensor array remains stationarywhile light one or more light filters are moved over the array to causecollection of the desired type of data. For example, in a first modetype 1 data are collected from the pixels in the presence or absence ofa light filter to remove type 2 data and in a second mode a filter toremove type 1 data while passing type 2 data are moved into position byMEMS over the array to collect type 2 data.

In yet another embodiment, the sensor array is moved and has differentlight filters positioned over one or more pixels in the array dependingon the operating mode. This embodiment is, effectively, a combination ofthe embodiments of FIGS. 13 and 17-18.

In yet another embodiment, the systems and methods of this disclosurecan be implemented in conjunction with a special purpose computer, aprogrammed microprocessor or microcontroller and peripheral integratedcircuit element(s), an ASIC or other integrated circuit, a digitalsignal processor, a hard-wired electronic or logic circuit such asdiscrete element circuit, a programmable logic device or gate array suchas PLD, PLA, FPGA, PAL, special purpose computer, any comparable means,or the like. In general, any device(s) or means capable of implementingthe methodology illustrated herein can be used to implement the variousaspects of this disclosure. Exemplary hardware that can be used for thedisclosed embodiments, configurations and aspects includes computers,handheld devices, telephones (e.g., cellular, Internet enabled, digital,analog, hybrids, and others), and other hardware known in the art. Someof these devices include processors (e.g., a single or multiplemicroprocessors), memory, nonvolatile storage, input devices, and outputdevices. Furthermore, alternative software implementations including,but not limited to, distributed processing or component/objectdistributed processing, parallel processing, or virtual machineprocessing can also be constructed to implement the methods describedherein.

In yet another embodiment, the disclosed methods may be readilyimplemented in conjunction with software using object or object-orientedsoftware development environments that provide portable source code thatcan be used on a variety of computer or workstation platforms.Alternatively, the disclosed system may be implemented partially orfully in hardware using standard logic circuits or VLSI design. Whethersoftware or hardware is used to implement the systems in accordance withthis disclosure is dependent on the speed and/or efficiency requirementsof the system, the particular function, and the particular software orhardware systems or microprocessor or microcomputer systems beingutilized.

In yet another embodiment, the disclosed methods may be partiallyimplemented in software that can be stored on a storage medium, executedon programmed general-purpose computer with the cooperation of acontroller and memory, a special purpose computer, a microprocessor, orthe like. In these instances, the systems and methods of this disclosurecan be implemented as program embedded on personal computer such as anapplet, JAVA® or CGI script or compiled software in a language such asC++ as a resource residing on a server or computer workstation, as aroutine embedded in a dedicated measurement system, system component, orthe like. The system can also be implemented by physically incorporatingthe system and/or method into a software and/or hardware system.

Although the present disclosure describes components and functionsimplemented in the aspects, embodiments, and/or configurations withreference to particular standards and protocols, the aspects,embodiments, and/or configurations are not limited to such standards andprotocols. Other similar standards and protocols not mentioned hereinare in existence and are considered to be included in the presentdisclosure. Moreover, the standards and protocols mentioned herein andother similar standards and protocols not mentioned herein areperiodically superseded by faster or more effective equivalents havingessentially the same functions. Such replacement standards and protocolshaving the same functions are considered equivalents included in thepresent disclosure.

The present disclosure, in various aspects, embodiments, and/orconfigurations, includes components, methods, processes, systems and/orapparatus substantially as depicted and described herein, includingvarious aspects, embodiments, configurations embodiments,subcombinations, and/or subsets thereof. Those of skill in the art willunderstand how to make and use the disclosed aspects, embodiments,and/or configurations after understanding the present disclosure. Thepresent disclosure, in various aspects, embodiments, and/orconfigurations, includes providing devices and processes in the absenceof items not depicted and/or described herein or in various aspects,embodiments, and/or configurations hereof, including in the absence ofsuch items as may have been used in previous devices or processes, e.g.,for improving performance, achieving ease and\or reducing cost ofimplementation.

The foregoing discussion has been presented for purposes of illustrationand description. The foregoing is not intended to limit the disclosureto the form or forms disclosed herein. In the foregoing DetailedDescription for example, various features of the disclosure are groupedtogether in one or more aspects, embodiments, and/or configurations forthe purpose of streamlining the disclosure. The features of the aspects,embodiments, and/or configurations of the disclosure may be combined inalternate aspects, embodiments, and/or configurations other than thosediscussed above. This method of disclosure is not to be interpreted asreflecting an intention that the claims require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive aspects lie in less than all features of a singleforegoing disclosed aspect, embodiment, and/or configuration. Thus, thefollowing claims are hereby incorporated into this Detailed Description,with each claim standing on its own as a separate preferred embodimentof the disclosure.

Moreover, though the description has included description of one or moreaspects, embodiments, and/or configurations and certain variations andmodifications, other variations, combinations, and modifications arewithin the scope of the disclosure, e.g., as may be within the skill andknowledge of those in the art, after understanding the presentdisclosure. It is intended to obtain rights which include alternativeaspects, embodiments, and/or configurations to the extent permitted,including alternate, interchangeable and/or equivalent structures,functions, ranges or steps to those claimed, whether or not suchalternate, interchangeable and/or equivalent structures, functions,ranges or steps are disclosed herein, and without intending to publiclydedicate any patentable subject matter.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. An apparatus comprising: (a) a sensor arraycomprising a plurality of Bayer patterns disposed on a first pluralityof pairs of adjacent linear arrays to sense Bayer pattern data, a secondplurality of pairs of adjacent linear arrays having different patternslocated between the first plurality of pairs of adjacent linear arraysto sense a different type of data; (b) an actuator to move the sensorarray from a first position to a second position, wherein physicalpositions of the first plurality of adjacent pairs of linear arraysalternate with physical positions of the second plurality of pairs ofadjacent linear arrays; and (c) circuitry coupled to the actuator andthe sensor array to output the Bayer pattern data and the different typeof data from each of the first position and the second position, thecircuitry comprising instructions to move the sensor array to the firstposition and measure first data of the sensor array in the firstposition and move the sensor array to the second position and measuresecond data of the sensor array in the second position, the processorfurther comprising instructions to provide a first full frame image fromthe first plurality of pairs of adjacent linear arrays having the Bayerpattern and to output a second full frame image from the secondplurality of pairs of adjacent linear arrays having the differentpatterns, wherein the sensor comprises a number of pixels and whereineach of the first full frame image and the second full frame imagecomprises the number of pixels of the sensor array.
 2. The apparatus inclaim 1, wherein each pixel of the Bayer pattern data is outputassociated with the first position or the second position and whereineach pixel of the different type of data is output associated with thefirst position or the second position.
 3. The apparatus in claim 1,wherein the different pattern comprises one or more of an infraredfilter pattern, an ultraviolet filter pattern, a non-Bayer visible lightfilter pattern, or a pattern comprising no filter.
 4. The apparatus inclaim 1, wherein each of the pairs of the first plurality of pairs ofthe adjacent linear arrays comprises two adjacent pairs and wherein eachof the second plurality of pairs of adjacent linear arrays comprises twoadjacent pairs of linear arrays.
 5. The apparatus in claim 1, whereineach of the pairs of the first plurality of pairs of the adjacent lineararrays comprises three or more adjacent pairs of linear arrays andwherein each of the second plurality of pairs of adjacent linear arrayscomprises three or more adjacent pairs of linear arrays.
 6. Theapparatus in claim 1, wherein the circuitry is configured to generatethe Bayer pattern data from first composite data and second compositedata and to generate the different type of data from the first compositedata and the second composite data.
 7. The apparatus in claim 1, whereinthe actuator comprises a micro-electrical mechanical system.
 8. Theapparatus in claim 1, wherein each of the plurality of Bayer patternscomprises a red pixel to sense red light, a blue pixel to sense bluelight, and a pair of diagonal green pixels to sense green light.
 9. Theapparatus in claim 1 further comprising digital data storage to store anoutput generated from the sensor arrays.
 10. The apparatus in claim 1further comprising a digital signal processor to control one or more ofthe following: transferring the output to digital data storage, timingof the image acquisition, movement of the arrays, configuring thecircuitry, configuring image formation, and generating images.